Where problems start

“The problem is that once there's ANY distance between your intention and your reward structure, bullshit slips in through that crack.” from some Facebook post I can’t be bothered to find again by Michael Smith (Morphenius) about incentives.
There are many examples of incentives that tend to reinforce collectives misaligned with their original intentions: companies that maximize profits instead of helping customers, charities that focus on pulling emotional strings instead of actually helping the cause they originally fought for, coaches that focus on milking customers instead of actually helping them solve their problems, etc.
Not only does this happen at all levels—individual to collective—but also in pretty much every domain. As Ran Prieur notes here: people often say that it isn’t money which is the root of all evils, but rather the love of money. But in fact, quote:

Money is totally intrinsically harmful. Whether or not you love it, money strips the meaning from physical items, from blocks of time, from human activities, and replaces those many diverse meanings with copies of the same meaning: this is a token which can be used to force other people to do shit.

So we could say that a vast class of problems occur when people lose track of what is actually valuable/meaningful and instead fall for the tokens we construct to organize our society. Money is the biggest one because it holds so much power, and we need it to survive in society, but it is not the only one: follower count is another major one of our times for instance.
This gap between intention and reward structure strikes me as incredibly fundamental but I feel like there is almost too much to say, so here are some crumbs for my future self to continue that train of thought if I am interested in that:

Collectives at a size significantly above Dunbar's number require complicated protocols/institutions to organize themselves -> this inevitably creates a gap between conscious intention and the reward structures of the protocols/institutions, because complicated structures cannot capture the entirety of a complex reality, which I would say is the deep reason behind Goodhart's law
Our world rewards people who are good at playing the game of rewards, not those who can stick to a conscious intention, as well as utterly destroying your relationship with what you personally want—school being the best example of that
The way people are manipulated is not through overt coercion or explicit propaganda, but by filtering who is allowed to have an impact in society and who isn't. This is the difference between disciplinary societies and societies of control, in Deleuze's terms
On an individual level, people burn out when their original intention has been replaced by a need to stick to protocols, or the need to conform to a shape that gets rewarded by external structures

Footnotes

Links and tags

Go back to the list of blog posts

Journal Untaggedsub

2024-08-05